2.1 Import data-set

First of all, we import the data-set with which we will work.

import pandas as pd

Netflix_Data = pd.read_csv('titles.csv')

Netflix_Data

2.2 Data-set conceptual description

This data-set has information about 15 variables on 5850 Netflix titles.

Next table has a brief conceptual description about data-set variables:

2.3 Data-set size

We can get the data-set size as the number of rows and columns of the data-set.

Netflix_Data.shape

(5850, 15)

As discussed above, the data-set has 5850 rows and 15 columns.

2.4 info() method

info() method give us column names, number of non null values in each column and column type.

Netflix_Data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5850 entries, 0 to 5849
Data columns (total 15 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   id                    5850 non-null   object 
 1   title                 5849 non-null   object 
 2   type                  5850 non-null   object 
 3   description           5832 non-null   object 
 4   release_year          5850 non-null   int64  
 5   age_certification     3231 non-null   object 
 6   runtime               5850 non-null   int64  
 7   genres                5850 non-null   object 
 8   production_countries  5850 non-null   object 
 9   seasons               2106 non-null   float64
 10  imdb_id               5447 non-null   object 
 11  imdb_score            5368 non-null   float64
 12  imdb_votes            5352 non-null   float64
 13  tmdb_popularity       5759 non-null   float64
 14  tmdb_score            5539 non-null   float64
dtypes: float64(5), int64(2), object(8)
memory usage: 685.7+ KB

2.5 Column types

There is another way to get column types.

Netflix_Data.dtypes

id                       object
title                    object
type                     object
description              object
release_year              int64
age_certification        object
runtime                   int64
genres                   object
production_countries     object
seasons                 float64
imdb_id                  object
imdb_score              float64
imdb_votes              float64
tmdb_popularity         float64
tmdb_score              float64
dtype: object

Object is the typical type of categorical variables, identifier or text.

Float64 and int64 is the typical type of quantitative variables, float64 for continuous one, and int64 for discrete one.

2.6 Change column types

We can change the type of a column with astype() method.

First, we check type of release_year variable is int64.

Netflix_Data.dtypes['release_year']

dtype('int64')

Now, we can change that type to float64 using the following function:

def change_type(Data, Variable_name, New_type):

    Data[Variable_name] = Data[Variable_name].astype(New_type)

change_type(Data=Netflix_Data, Variable_name='release_year', New_type='float64')

We can check if the changes have been done correctly:

Netflix_Data.dtypes['release_year']

dtype('float64')

2.7 Unique values of a variable

We can get the uniques values of a variable with the unique() method.

We can get the uniques values of type as following :

Netflix_Data['type'].unique()

array(['SHOW', 'MOVIE'], dtype=object)

\(\\\)

We can get the unique values of age_certification as following :

Netflix_Data['age_certification'].unique()

array(['TV-MA', 'R', 'PG', nan, 'TV-14', 'PG-13', 'TV-PG', 'TV-Y', 'TV-G', 'TV-Y7', 'G', 'NC-17'], dtype=object)

\(\\\)

We can get the uniques values of production_countries as following :

Netflix_Data['production_countries'].unique()     

array(["['US']", "['GB']", "['GB', 'US']", "['EG']", "['DE']", "['IN']",
       "['SU', 'IN']", "['LB', 'CA', 'FR']", '[]', "['LB']",
       "['DZ', 'EG']", "['CA', 'FR', 'LB']", "['US', 'GB']",
       "['US', 'IT']", "['JP']", "['AR']", "['FR', 'EG']", "['FR', 'LB']",
       "['CA', 'US']", "['US', 'FR']", "['JP', 'US']", "['US', 'CA']",
       "['DE', 'US']", "['PE', 'US', 'BR']", "['IT', 'US', 'FR']",
       "['IE', 'GB', 'DE', 'FR']", "['HK', 'US']", "['AU']", "['FR']",
       "['DE', 'GH', 'GB', 'US', 'BF']", "['MX']", "['ES', 'AR']",
       "['CO']", "['PS', 'US', 'FR', 'DE']", "['FR', 'NO', 'LB', 'BE']",
       "['BE', 'FR', 'IT', 'LB']", "['TR']", "['IN', 'SU']", "['DK']",
       "['CA']", "['DE', 'GB', 'US', 'BS', 'CZ']", "['MT', 'GB', 'US']",
       "['AU', 'DE', 'GB', 'US']", "['US', 'JP']", "['BE', 'US']",
       "['HK']", "['IT']", "['US', 'FR', 'DE', 'GB']",
       "['GB', 'US', 'FR', 'DE']", "['IT', 'US']", "['US', 'ZA']",
       "['GB', 'ES']", "['GB', 'US', 'JP']", "['HK', 'CN']",
       "['GB', 'US', 'BG']", "['RU']", "['KR']", "['CA', 'US', 'IN']",
       "['CN']", "['JP', 'HK']", "['CA', 'GB', 'US']",
       "['FR', 'MX', 'ES']", "['IN', 'US']", "['AR', 'ES']", "['CL']",
       "['FR', 'MA', 'DE', 'PS']", "['AR', 'DE', 'UY', 'ES']",
       "['CL', 'AR']", "['CZ', 'GB', 'DK', 'NL', 'SE']", "['TW']",
       "['SG']", "['NG']", "['MY']", "['Lebanon']",
       "['BE', 'FR', 'ES', 'CH', 'PS']", "['ZA']", "['NG', 'US']",
       "['LB', 'FR']", "['CN', 'HK']", "['PH']", "['LB', 'GB', 'FR']",
       "['FR', 'DE', 'KW', 'PS']", "['PS']",
       "['GB', 'US', 'AT', 'FR', 'DE', 'NG']", "['XX']", "['AE', 'US']",
       "['DK', 'US']", "['FR', 'US', 'GB']", "['HU', 'US', 'CA']",
       "['NO']", "['GB', 'FR', 'DE']", "['US', 'HU', 'IT']",
       "['US', 'ZA', 'DE']", "['IN', 'DE']", "['SA']", "['ID']",
       "['US', 'LB', 'AE']", "['PS', 'NL', 'US', 'AE']",
       "['US', 'FR', 'GB']", "['US', 'DE', 'GB']", "['GB', 'ZA']",
       "['US', 'CA', 'CL']", "['US', 'GB', 'CN', 'CA']",
       "['AU', 'CH', 'GB']", "['ES']", "['FI']", "['IL']", "['FR', 'US']",
       "['AU', 'US']", "['CA', 'US', 'GB']", "['AT']", "['CD', 'GB']",
       "['US', 'BR']", "['CA', 'JP', 'US']", "['CA', 'KR']",
       "['US', 'EG', 'GB']", "['BR']", "['PL']", "['VE', 'AR']", "['RO']",
       "['IL', 'NO', 'ZA', 'AE', 'GB', 'IS', 'IE']",
       "['US', 'CN', 'DE', 'SG', 'UA']", "['DE', 'IT', 'PS', 'FR']",
       "['AE', 'LB']", "['LB', 'AE']", "['US', 'ES']", "['NZ']",
       "['GB', 'US', 'FR']", "['US', 'FR', 'LU', 'GB']", "['FR', 'BE']",
       "['IT', 'GB']", "['US', 'CA', 'GB']", "['CA', 'FR']",
       "['US', 'CN']", "['UA']", "['MX', 'ZA', 'US']",
       "['US', 'GB', 'ES']", "['BE', 'DK', 'DE', 'GB', 'US']",
       "['GB', 'IR', 'JO', 'QA']", "['CH', 'US']", "['CA', 'DE', 'GB']",
       "['GH', 'US']", "['IE', 'GB']", "['CN', 'US']",
       "['UA', 'GB', 'US']", "['IE', 'ZA']", "['US', 'FR', 'MT']",
       "['BG']", "['GB', 'FR']", "['BY']", "['IE']", "['IS']",
       "['AU', 'FR', 'DE']", "['CN', 'FR', 'CA']", "['FR', 'QA']",
       "['SE']", "['FR', 'ES']", "['NL']", "['HR']", "['FR', 'MA']",
       "['RU', 'US', 'FR']", "['SY', 'GB']", "['AT', 'US']", "['CD']",
       "['FR', 'CL']", "['AU', 'GB']", "['TN']", "['AE']", "['SE', 'NO']",
       "['GL', 'FR']", "['LB', 'DE']", "['PT', 'SE', 'DK', 'BR', 'FR']",
       "['QA', 'LB']", "['GB', 'AU', 'US']", "['ES', 'DK']",
       "['AE', 'FR', 'JO', 'LB', 'QA', 'PS']", "['US', 'CA', 'JP']",
       "['PK']", "['IN', 'GB']", "['PS', 'FR', 'DE']", "['CZ']",
       "['CA', 'NG']", "['VN']", "['NL', 'GB']",
       "['CA', 'HU', 'MX', 'ES', 'GB', 'US']", "['FR', 'GB', 'US']",
       "['FR', 'NL', 'GB', 'US']", "['CN', 'CA', 'US']", "['CA', 'GB']",
       "['KR', 'US']", "['FR', 'RO', 'GB', 'BE', 'DE']", "['US', 'MX']",
       "['HK', 'IS', 'US']", "['IN', 'CN', 'US', 'GB']", "['BE', 'FR']",
       "['PR', 'US', 'GB', 'CN']", "['GB', 'DE']", "['US', 'PR']",
       "['IT', 'CH', 'FR']", "['IT', 'ES', 'FR']", "['US', 'IS', 'NO']",
       "['IQ', 'GB']", "['HU']", "['US', 'AU', 'GB']",
       "['CZ', 'GB', 'US']", "['US', 'IE', 'CA']", "['TH']",
       "['IR', 'US', 'FR']", "['BE']",
       "['GB', 'ID', 'CA', 'CN', 'SG', 'US']", "['ES', 'FR']",
       "['SG', 'GB', 'US']", "['GE', 'DE', 'FR']", "['CA', 'US', 'DE']",
       "['CA', 'IE']", "['NL', 'BE']", "['US', 'KH']", "['FR', 'JP']",
       "['PR']", "['US', 'CA', 'CN']", "['CN', 'US', 'ES']",
       "['CU', 'US']", "['BG', 'US']", "['US', 'BG']",
       "['US', 'DK', 'GB']", "['ES', 'IT']", "['TR', 'US']",
       "['PE', 'DE', 'NO']", "['LU', 'US', 'FR']",
       "['IL', 'MA', 'US', 'BG', 'GB']", "['AR', 'CL']",
       "['AR', 'ES', 'UY']", "['JP', 'CN']", "['US', 'AU']",
       "['QA', 'TN', 'FR']", "['ES', 'MX']", "['PH', 'SG']",
       "['US', 'AE']", "['DE', 'DK', 'NL', 'GB']", "['NL', 'MX']",
       "['CA', 'CN']", "['NO', 'SE', 'DK', 'NL']", "['US', 'DE', 'ZA']",
       "['IS', 'SE', 'BE']", "['DE', 'ES']", "['CN', 'FR', 'TW', 'US']",
       "['KH']", "['BE', 'FR', 'IT']", "['DE', 'CH']",
       "['JP', 'KR', 'FR']", "['DE', 'NZ', 'GB']", "['PE']",
       "['MX', 'US']", "['US', 'DK']", "['PL', 'US']", "['KE']", "['GH']",
       "['IT', 'CH', 'VA', 'FR', 'DE']", "['PE', 'GB', 'US', 'IL', 'IT']",
       "['SA', 'SY', 'AE']", "['US', 'KR']", "['IN', 'FR']",
       "['RS', 'PL', 'RU']", "['CL', 'NL', 'FR']", "['IE', 'CA']",
       "['US', 'NL']", "['TZ']", "['IT', 'ES']", "['ID', 'MY', 'SG']",
       "['FR', 'LU', 'CA']", "['FR', 'QA', 'TN', 'BE']",
       "['PL', 'CH', 'AL', 'IT']", "['CZ', 'US']", "['AR', 'FR']",
       "['DE', 'IT']", "['IT', 'FR']", "['MX', 'FI']", "['CA', 'BR']",
       "['IN', 'MX']", "['BR', 'DK', 'FR', 'DE', 'PL', 'AR']",
       "['ZA', 'US', 'CA']", "['ES', 'BE']", "['PY']", "['US', 'NG']",
       "['US', 'BE', 'GB']", "['ZW']", "['IT', 'AR']",
       "['AT', 'IQ', 'US']", "['GE']", "['AR', 'IT']", "['NG', 'NO']",
       "['IS', 'GB']", "['MX', 'CO']", "['AR', 'US']", "['KW']",
       "['JP', 'GB']", "['TW', 'US']", "['NP', 'IN']",
       "['AU', 'US', 'CN']", "['FR', 'IN', 'SG']", "['LB', 'PS']",
       "['JP', 'US', 'CA']", "['CM']", "['BD', 'IN']", "['CA', 'ZA']",
       "['FR', 'PS', 'CH', 'QA']", "['NL', 'JO', 'DE']",
       "['GB', 'DK', 'GR']", "['MX', 'AR']", "['US', 'CL', 'MX']",
       "['KG']", "['CH']", "['BD']", "['LU']", "['ZA', 'GB']",
       "['BT', 'CN']", "['CA', 'HU', 'US']", "['BE', 'LT', 'NL']",
       "['IT', 'MC', 'US', 'CA']", "['CN', 'US', 'AU', 'CA']",
       "['BE', 'SE', 'GB']", "['GB', 'CZ', 'FR']", "['US', 'MW', 'GB']",
       "['US', 'CY']", "['BE', 'FR', 'SN']", "['BR', 'FR', 'ES', 'BE']",
       "['US', 'CH']", "['US', 'IL']", "['FR', 'LT', 'GB']",
       "['GB', 'IE']", "['GB', 'IT']", "['JO', 'TH', 'US', 'AL']",
       "['PT', 'US']", "['IL', 'US', 'FR', 'DE']", "['TW', 'MY']",
       "['US', 'CA', 'FR', 'ES']", "['FI', 'NO']", "['US', 'FR', 'JP']",
       "['GB', 'JP']", "['US', 'CN', 'GB']",
       "['US', 'FR', 'SE', 'GB', 'DE', 'DK', 'CA']", "['DE', 'AT']",
       "['US', 'TH']", "['PH', 'US']", "['BR', 'MX']", "['NO', 'CA']",
       "['CO', 'ES']", "['CN', 'DE', 'GB']", "['NO', 'DE']",
       "['ES', 'PT']", "['IL', 'US']", "['ES', 'BE', 'DE']",
       "['TH', 'US']", "['US', 'FR', 'ES']", "['ES', 'FR', 'AR']",
       "['NL', 'PL', 'UA', 'GB', 'US']", "['QA', 'PS']",
       "['RS', 'UY', 'AR']", "['FR', 'IT']", "['CA', 'LK']",
       "['US', 'AR']", "['EG', 'US']", "['US', 'IN']",
       "['FR', 'LU', 'BE', 'KH']", "['US', 'BE', 'ES']",
       "['CA', 'FR', 'JP', 'GB', 'US']", "['AT', 'DE']",
       "['US', 'GB', 'DE']", "['FR', 'MX', 'CO']", "['BR', 'FR']",
       "['JO']", "['FR', 'IN', 'QA']", "['AR', 'PE']", "['MU']",
       "['DE', 'DK', 'EG']", "['US', 'IE']", "['IO']", "['TW', 'CN']",
       "['FR', 'NL', 'SG']", "['SN']", "['UY']", "['DE', 'IN', 'AT']",
       "['MA', 'FR', 'QA']", "['PS', 'PH']", "['EG', 'SA']",
       "['ES', 'CN']", "['CL', 'AR', 'CA']", "['AR', 'CO']",
       "['GT', 'UY']", "['AF', 'DE', 'PS']", "['ZA', 'AO']",
       "['HK', 'PH']", "['SG', 'MY']", "['SE', 'US']",
       "['LB', 'US', 'NL', 'CA']", "['NL', 'PS', 'US', 'LB']",
       "['DK', 'LB', 'GB']", "['UY', 'MX', 'ES']", "['PH', 'JP']",
       "['CN', 'JP', 'US']", "['NA']", "['LB', 'QA', 'SY', 'FR']",
       "['PS', 'DK', 'LB']", "['US', 'CZ']",
       "['GB', 'AU', 'CA', 'GR', 'NZ']", "['GR', 'GB', 'US']",
       "['DE', 'FR']", "['NL', 'US']", "['AT', 'GB', 'US']",
       "['CH', 'DE']", "['GB', 'US', 'DE']", "['DK', 'IS']",
       "['FR', 'DE', 'US']", "['US', 'JP', 'TH']", "['FR', 'DE']",
       "['RO', 'US']", "['ES', 'KN']", "['SE', 'GB']",
       "['SG', 'US', 'IN']", "['DE', 'AU']", "['GB', 'CA']",
       "['IE', 'US', 'CA']", "['PT']", "['US', 'PL', 'KR']",
       "['LU', 'FR']", "['IT', 'BR']", "['GB', 'HU', 'NL', 'CH']",
       "['BR', 'DE', 'QA', 'MX', 'US', 'CH', 'AR']", "['ES', 'PE']",
       "['BE', 'GB', 'DE']", "['ZA', 'GB', 'US']", "['CL', 'PE']",
       "['CA', 'CN', 'US']", "['SG', 'US']", "['BR', 'US']",
       "['BE', 'NL']", "['RU', 'US']", "['ES', 'US']", "['CZ', 'DE']",
       "['NZ', 'HK']", "['MA', 'SA', 'TN', 'EG', 'LB']", "['CN', 'GB']",
       "['AF']", "['BE', 'LU']", "['BE', 'DE']", "['SE', 'RO']",
       "['ZA', 'US']", "['GB', 'IN']", "['HU', 'CA']", "['NG', 'CA']",
       "['TZ', 'GB']", "['PH', 'FO']"], dtype=object)

2.8 NaN identification

A NaN is a not a number value. NaN is equivalent to missing value.

We are going to calculate, for each variable, the proportion of missing values over the total number of observations. We can do it using isnull() method :

def Prop_NaN(Data):

    df_prop_nan = Data.isnull().sum() / len(Data)

    return df_prop_nan

Prop_NaN(Data=Netflix_Data)

id                      0.000000
title                   0.000171
type                    0.000000
description             0.003077
release_year            0.000000
age_certification       0.447692
runtime                 0.000000
genres                  0.000000
production_countries    0.000000
seasons                 0.640000
imdb_id                 0.068889
imdb_score              0.082393
imdb_votes              0.085128
tmdb_popularity         0.015556
tmdb_score              0.053162
dtype: float64

We can see that there are variables with a high proportion of missing values, as age_certification (44.77%).

season would be the variable with higher proportion of missing values, but it is because of season only is defined for type=SHOW.

2.9 Variable Scaling

Scaling a variable is applying a transformation, in order to obtain new properties for the transformed variable, properties that the original variable doesn’t have.

In this article, we will focus on three scaling methods: standard scaling, normalization (0,1), and normalization (a,b).

In any case, there are more procedures that will not be explored here, so for a more extensive list, it is recommended to consult the sklearn documentation on this topic: https://scikit-learn.org/stable/modules/preprocessing.html

Some of the concepts that appear in this secction will be explained with more details in Statistical Description section, such as the concept of statistical variable, sample, mean and variance.

from sklearn import preprocessing

def standard_scaling(Data, Variable_name):

    scaler = preprocessing.StandardScaler().fit(Data[[Variable_name]])

    Data[Variable_name + '_std_scaling'] = scaler.transform(Netflix_Data[[Variable_name]])

standard_scaling(Data=Netflix_Data, Variable_name='release_year')

Netflix_Data[['release_year', 'release_year_std_scaling']]

Netflix_Data['release_year_std_scaling'].mean()

-1.010549668636587e-14

Netflix_Data['release_year_std_scaling'].std()

1.0000854810447344

def Normalization(Data, Variable_name, min, max):

    min_max_scaler = preprocessing.MinMaxScaler(feature_range=(min,max))

    min = str(min)
    max = str(max)

    Data[Variable_name + '_norm_' + min + '_' + max] = min_max_scaler.fit_transform(Data[[Variable_name]])

Normalization(Data=Netflix_Data, Variable_name='release_year', min=0, max=1)

Netflix_Data[['release_year', 'release_year_norm_0_1']]

Netflix_Data['release_year_norm_0_1'].min()

0.0

Netflix_Data['release_year_norm_0_1'].max()

1.0

Normalization(Data=Netflix_Data, Variable_name='release_year', min=2, max=7)

Netflix_Data[['release_year', 'release_year_norm_2_7']]

Netflix_Data['release_year_norm_2_7'].min()

2.0

Netflix_Data['release_year_norm_2_7'].max()

6.999999999999986

2.10 Standard Recoding of Categorical Variables

Given a categorical statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\) with range \(\hspace{0.07cm}R( X_k) = \lbrace g_1, g_2 , ..., g_h \rbrace\hspace{0.07cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

Properties :

• \(\hspace{0.1cm}R( X_k^{cod}) = \lbrace 0,1,..., h-1 \rbrace\)

from sklearn.preprocessing import OrdinalEncoder

def Standard_recoding(Data, Variable_name) :

    Data[Variable_name + '_recode'] = OrdinalEncoder().fit_transform(Data[[Variable_name]])

Standard_recoding(Data=Netflix_Data , Variable_name = 'type')

Netflix_Data.loc[ : , ['type','type_recode']].head()

Standard_recoding(Data=Netflix_Data , Variable_name = 'age_certification')

df1 = pd.DataFrame()

for i in range(0,11):

    df2 = Netflix_Data.loc[ Netflix_Data['age_certification_recode'] == i , ['age_certification','age_certification_recode']  ].head(1)

    df1 = pd.concat([df1 , df2], axis=0)

df1

2.11 Categorization of Quantitative Variables

Given a quantitative statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

• ¿How to define the categorization intervals?

  There are different procedures to define the categorization intervals. Some of the most common are quantile-based rules.

  Next, we are going to expose some procedures based on quantiles and another alternative, Scott’s rule.

pd.cut(x=Netflix_Data['release_year'] , bins=5 )

0       (1944.923, 1960.4]
1         (1975.8, 1991.2]
2         (1960.4, 1975.8]
3         (1960.4, 1975.8]
4         (1960.4, 1975.8]
               ...        
5845      (2006.6, 2022.0]
5846      (2006.6, 2022.0]
5847      (2006.6, 2022.0]
5848      (2006.6, 2022.0]
5849      (2006.6, 2022.0]
Name: release_year, Length: 5850, dtype: category
Categories (5, interval[float64, right]): [(1944.923, 1960.4] < (1960.4, 1975.8] < (1975.8, 1991.2] < (1991.2, 2006.6] < (2006.6, 2022.0]]

pd.cut(x=Netflix_Data['release_year'] , bins=5 , labels=False)

0       0
1       2
2       1
3       1
4       1
       ..
5845    4
5846    4
5847    4
5848    4
5849    4
Name: release_year, Length: 5850, dtype: int64

intervals = [Netflix_Data['release_year'].min() - 0.5 , Netflix_Data['release_year'].median(), Netflix_Data['release_year'].max() ]
intervals

[1944.5, 2018.0, 2022.0]

pd.cut(x=Netflix_Data['release_year'], bins=intervals )

0       (1944.5, 2018.0]
1       (1944.5, 2018.0]
2       (1944.5, 2018.0]
3       (1944.5, 2018.0]
4       (1944.5, 2018.0]
              ...       
5845    (2018.0, 2022.0]
5846    (2018.0, 2022.0]
5847    (2018.0, 2022.0]
5848    (2018.0, 2022.0]
5849    (2018.0, 2022.0]
Name: release_year, Length: 5850, dtype: category
Categories (2, interval[float64, right]): [(1944.5, 2018.0] < (2018.0, 2022.0]]

pd.cut(x=Netflix_Data['release_year'], bins=intervals, labels=False )

0       0
1       0
2       0
3       0
4       0
       ..
5845    1
5846    1
5847    1
5848    1
5849    1
Name: release_year, Length: 5850, dtype: int64

intervals = [Netflix_Data['release_year'].min() - 1 , Netflix_Data['release_year'].quantile(0.25), Netflix_Data['release_year'].quantile(0.5), Netflix_Data['release_year'].quantile(0.75), Netflix_Data['release_year'].max()]
intervals

[1944.0, 2016.0, 2018.0, 2020.0, 2022.0]

Netflix_Data['release_year_cat_int_1'] = pd.cut(x=Netflix_Data['release_year'], bins=intervals )

Netflix_Data['release_year_cat_1'] = pd.cut(x=Netflix_Data['release_year'], bins=intervals, labels=False )

Netflix_Data.loc[: , ['release_year','release_year_cat_int_1','release_year_cat_1']]

import numpy as np

intervals = []

for q in np.arange(0, 1, step=0.1) :

    intervals.append( Netflix_Data['imdb_score'].quantile(q))

np.arange(0, 1, step=0.1)

array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])

intervals

[1.5, 5.0, 5.6, 6.0, 6.3, 6.6, 6.9, 7.2, 7.5, 7.9]

intervals[0] = intervals[0] - 0.5 # El primer extremo debe ser Q(0) - c

intervals.append(Netflix_Data['imdb_score'].quantile(1)) # El ultimo extremo debe ser Q(1)

intervals

[1.0, 5.0, 5.6, 6.0, 6.3, 6.6, 6.9, 7.2, 7.5, 7.9, 9.6]

Netflix_Data['imdb_score_cat_int_1'] = pd.cut(x=Netflix_Data['imdb_score'], bins=intervals )

Netflix_Data['imdb_score_cat_1'] = pd.cut(x=Netflix_Data['imdb_score'], bins=intervals, labels=False )

Netflix_Data.loc[: , ['imdb_score','imdb_score_cat_int_1','imdb_score_cat_1']]

df1 = pd.DataFrame()

for i in range(0,len(Netflix_Data['imdb_score_cat_1'].unique())):

    df2 = Netflix_Data.loc[ Netflix_Data.imdb_score_cat_1 == i , ['imdb_score_cat_int_1','imdb_score_cat_1']  ].head(1)

    df1 = pd.concat([df1 , df2], axis=0)

df1

def Categorization_quantiles_rule(Data, Variable_name, t):

    intervals = []

    for q in np.arange(0, 1, step=t) :

        intervals.append( Data[Variable_name].quantile(q))

    intervals[0] = intervals[0] - 0.5

    intervals.append(Data[Variable_name].quantile(1))

    Data[Variable_name + '_cat_interval'] = pd.cut(x=Data[Variable_name], bins=intervals)

    Data[Variable_name + '_cat'] = pd.cut(x=Data[Variable_name], bins=intervals, labels=False) 
 

Categorization_quantiles_rule(Data=Netflix_Data, Variable_name='imdb_score', t=0.05)

Netflix_Data.loc[: , ['imdb_score','imdb_score_cat_int','imdb_score_cat']]

df1 = pd.DataFrame()

for i in range(0,len(Netflix_Data['imdb_score_cat_2'].unique())):

    df2 = Netflix_Data.loc[ Netflix_Data.imdb_score_cat == i , ['imdb_score_cat_int_2','imdb_score_cat_2']  ].head(1)

    df1 = pd.concat([df1 , df2], axis=0)

df1 

Categorization_quantiles_rule(Data=Netflix_Data, Variable_name='imdb_score', t=0.25)

Netflix_Data.loc[: , ['imdb_score','imdb_score_cat_interval','imdb_score_cat']]

df1 = pd.DataFrame()

for i in range(0,len(Netflix_Data['imdb_score_cat'].unique())):

    df2 = Netflix_Data.loc[ Netflix_Data.imdb_score_cat == i , ['imdb_score_cat_interval','imdb_score_cat']  ].head(1)

    df1 = pd.concat([df1 , df2], axis=0)

df1

Categorization_quantiles_rule(Data=Netflix_Data, Variable_name='imdb_score', t=0.1)

Netflix_Data.loc[: , ['imdb_score','imdb_score_cat_interval','imdb_score_cat']]

df1 = pd.DataFrame()

for i in range(0,len(Netflix_Data['imdb_score_cat'].unique())):

    df2 = Netflix_Data.loc[ Netflix_Data.imdb_score_cat == i , ['imdb_score_cat_interval','imdb_score_cat']  ].head(1)

    df1 = pd.concat([df1 , df2], axis=0)

df1

2.12 Dummification of Categorical Variables

Given a categorical statistical variable \(\hspace{0.06cm}\mathcal{X}_k\hspace{0.05cm}\) with range \(\hspace{0.06cm}R(\mathcal{X}_k)=\lbrace g_1 ,..., g_h \rbrace\hspace{0.06cm}\) , and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

Dummify \(\hspace{0.06cm}X_k\hspace{0.06cm}\) consist of obtaining the new dummy samples \(\hspace{0.06cm}X_{k\hspace{0.06cm}0}\hspace{0.06cm},\hspace{0.06cm}X_{k\hspace{0.06cm}1}\hspace{0.06cm},...,\hspace{0.06cm}X_{k \hspace{0.06cm}h-1}\hspace{0.06cm}\), where \(\hspace{0.06cm}X_{kj}\hspace{0.06cm}\) is defined as:

\[x_{i,kj} = \left\lbrace\begin{array}{l} 1 \hspace{0.3cm} , \hspace{0.3cm} \text{ if} \hspace{0.3cm} x_{ik} = j \\[0.1cm] 0 \hspace{0.25cm}, \hspace{0.3cm} \text{ if} \hspace{0.3cm} x_{ik} \neq j \end{array}\right.\]

for \(\hspace{0.06cm}j\in \lbrace 0\hspace{0.06cm},\hspace{0.06cm}1\hspace{0.06cm},\dots ,\hspace{0.06cm}h-1 \rbrace\)

def dummies(Data, Variable_name, drop_first=False):

    df_dummies = pd.get_dummies(Data[Variable_name], drop_first=drop_first)

    return df_dummies

dummies(Data=Netflix_Data, Variable_name='type')

dummies(Data=Netflix_Data, Variable_name='age_certification')

dummies(Data=Netflix_Data, Variable_name='age_certification', drop_first=True)

2.13 Dealing with NaN

3.1 Statistical variable

A statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\) can be modeled as a random variable.

Under this approach, we can apply all probability theory on random variables to statistical variables. \(\\[0.4cm]\)

3.2 Range of a statistical variable

The range of a statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\) is denoted by \(\hspace{0.05cm}Range(\mathcal{X}_k)\hspace{0.05cm}\), and is defined as the set of possible values of \(\hspace{0.05cm}\mathcal{X}_k\). \(\\[0.4cm]\)

3.3 Sample of a statistical variable

Given a statistical variable \(\hspace{0.05cm}\mathcal{X}_k\).

A sample of \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\) is a vector of values of \(\hspace{0.05cm}\mathcal{X}_k\), called observations.

Therefore:

\[ X_k \hspace{0.05cm} = \hspace{0.05cm} \begin{pmatrix} x_{1k} \\ x_{2k}\\ ... \\ x_{nk} \end{pmatrix} \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t \\ \]

is a sample of a statistical variable because is a vector with the values or observations of the variable \(\hspace{0.05cm} \mathcal{X}_k \hspace{0.05cm}\) for \(\hspace{0.05cm} n \hspace{0.05cm}\) elements or individuals of a sample.

Where: \(\hspace{0.1cm} x_{ik}\hspace{0.05cm}\) is the value \(\hspace{0.05cm} i\)-th observation of the variable \(\hspace{0.05cm} \mathcal{X}_k\). \(\\[0.4cm]\)

3.4 Arithmetic Mean

Given a quantitative statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

The arithmetic mean of \(\hspace{0.05cm}X_k \hspace{0.05cm}\) is defined as: \(\\[0.3cm]\)

\[\overline{\hspace{0.05cm} X_k \hspace{0.05cm} } \hspace{0.1cm}=\hspace{0.1cm} \dfrac{1}{n} \cdot \sum_{i=1}^n \hspace{0.05cm} x_{ik}\] \(\\[0.4cm]\)

Properties:

• Existence: the arithmetic mean of a sample \(X_k\) of a statisitcal variable \(\mathcal{X}_k\) always exist, for any \(X_{k}\).

• Commutatividad: arithmetic mean isn’t affected by the order of the elements of the sample \(X_k\) .

• \(\overline{X_k} + \overline{X_j} = \overline{X_k + X_j}\)

• \(\overline{ a\cdot X_k + b} = a \cdot \overline{X_k} + b\) , for any \(a,b \in \mathbb{R}\)

3.5 Weighted Mean

Given a quantitative statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

And given a weights for each observation of the variable \(\hspace{0.05cm} \mathcal{X}_k \hspace{0.2cm} \Rightarrow \hspace{0.2cm}\) \(w \hspace{0.05cm} = \hspace{0.05cm} (w_1,w_2,...,w_n)^t\)

The weighted mean of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) with the weights vector \(\hspace{0.05cm} w \hspace{0.05cm}\) is defined as:

\[ \overline{X_k} (w) \hspace{0.1cm} = \hspace{0.1cm} \dfrac{1}{\hspace{0.1cm}\sum_{i=1}^{n} \hspace{0.05cm} w_{i} \hspace{0.1cm}} \hspace{0.05cm}\cdot\hspace{0.05cm} \sum_{i=1}^{n} \hspace{0.1cm} x_{ik} \cdot w_i \] \(\\[0.4cm]\)

3.6 Geometric Mean

Given the variable \(\hspace{0.05cm} X_k=(x_{1k}, x_{2k},...,x_{nk})^t\).

The geometric mean of the variable \(\hspace{0.05cm}X_k\hspace{0.05cm}\) is defined as: \(\\[0.3cm]\)

\[ \overline{X_k}_{geo} \hspace{0.05cm} = \hspace{0.05cm} \sqrt{\Pi_{i=1}^{n} x_{ik}} \hspace{0.05cm} = \hspace{0.05cm} \sqrt{x_{1k}\cdot x_{2k}\cdot...\cdot x_{nk}} \] \(\\[0.4cm]\)

3.7 Median

Given a statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

The median of \(\hspace{0.05cm}X_k \hspace{0.05cm}\) is defined as a value \(Me(X_k)\) such that: \(\\[0.3cm]\)

\[\dfrac{1}{n} \cdot \sum_{i=1}^n \hspace{0.1cm} \mathbb{I} \hspace{0.05cm} \bigl[ \hspace{0.1cm} x_{ik} \hspace{0.05cm} \leq \hspace{0.05cm} Me(X_k) \hspace{0.1cm} \bigr] \hspace{0.1cm} = \hspace{0.1cm} 0.50\]

where: \(\hspace{0.15cm}\mathbb{I}\hspace{0.1cm}\) is the indicator function. \(\\[0.4cm]\)

Properties:

Existencia: La mediana siempre existe para cualquier conjunto de números.

Invariante a permutaciones: El orden de los números no afecta a la mediana.

No linealidad: La mediana de una suma de números no es igual a la suma de las medianas de cada conjunto de números.

Invarianza a la escala: Multiplicar todos los números por una constante no afecta la mediana.

Si se cumple \(median(cX_j) = c\cdot median(X_j)\) pero no es cierto en general que \(median(cX_j + b) = c\cdot median(X_j) + b\)

3.8 Mode

Given a categorical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

The mode of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is the most repeated value in \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\), so, the mode of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is the most frequent value of \(\hspace{0.05cm} X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) \(\\[0.4cm]\)

3.9 Variance

Given a quantitative variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

The variance of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is defined as:

\[\sigma(X_k)^2 \hspace{0.1cm} = \hspace{0.1cm} \dfrac{1}{n} \cdot \sum_{i=1}^n \hspace{0.05cm} \left(\hspace{0.05cm} x_{ik} - \overline{X_k} \hspace{0.05cm}\right)^2\]

The standard deviation or standard error of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is defined as:

\[\sqrt{ \sigma(X_k)^2 } \hspace{0.1cm} = \hspace{0.1cm} \dfrac{1}{n} \cdot \sum_{i=1}^n \left( \hspace{0.05cm} x_{ik} - \overline{X_k} \hspace{0.05cm} \right)\] \(\\[0.4cm]\)

3.10 Median Absolute Deviation

Given a statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

The median absolute deviation (MAD) of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is defined as:

\[MAD(X_k) \hspace{0.1cm} = \hspace{0.1cm} Me \bigl( \hspace{0.1cm} \left| \hspace{0.05cm} X_k - Me(X_k) \hspace{0.05cm} \right| \hspace{0.1cm} \bigr) \hspace{0.1cm} = \hspace{0.1cm} Me \hspace{0.1cm} \Bigr[ \hspace{0.1cm} \left( \hspace{0.2cm} \left| \hspace{0.1cm} x_{ik} - Me(X_k) \hspace{0.1cm} \right| \hspace{0.15cm} : \hspace{0.15cm} i = 1,\dots,n \hspace{0.2cm} \right) \hspace{0.1cm} \Bigr]\] \(\\[0.4cm]\)

3.11 Quantiles

Given a statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

The \(\hspace{0.05cm}q\)-order quantile of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is defined as a value \(Q(X_k , q)\) such that:

\[\dfrac{1}{n} \cdot \sum_{i=1}^n \hspace{0.1cm} \mathbb{I} \hspace{0.05cm} \bigl[ \hspace{0.1cm} x_{ik} \hspace{0.05cm} \leq \hspace{0.05cm} Q(\hspace{0.05cm} X_k \hspace{0.05cm},\hspace{0.05cm} q \hspace{0.05cm}) \hspace{0.1cm} \bigr] \hspace{0.1cm} = \hspace{0.1cm} q\]

where: \(\hspace{0.15cm}\mathbb{I}\hspace{0.1cm}\) is the indicator function. \(\\[0.3cm]\)

Observation:

The median is the 0.5-order quantile. \(\\[0.4cm]\)

3.12 Kurtosis

Given a quantitative variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

The kurtosis coefficient of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is defined as: \(\\[0.35cm]\)

\[ \Psi(X_k) = \dfrac{\mu_{4}}{\sigma(X_k)^{4}} \]

where:

\[ \mu_{4}\hspace{0.1cm} =\hspace{0.1cm} \frac{1}{n} \cdot \sum_{i=1}^{n} \hspace{0.05cm} x_{ik}^4 \\[0.3cm] \]

Propierties:

• If \(\hspace{0.12cm}\Psi(X_k) \hspace{0.05cm} > \hspace{0.05cm} 3\hspace{0.08cm}\) \(\hspace{0.2cm}\Rightarrow\hspace{0.2cm}\) the distribution of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is more pointed and with longer tails than the normal distribution. \(\\[0.5cm]\)

• If \(\hspace{0.12cm}\Psi(X_k) \hspace{0.05cm} < \hspace{0.05cm} 3\hspace{0.08cm}\) \(\hspace{0.2cm}\Rightarrow\hspace{0.2cm}\) the distribution of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is less pointed and with shorter tails than the normal distribution. \(\\[0.4cm]\)

3.13 Skewness

Given a quantitative variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

The skewness coefficient of \(\hspace{0.05cm} X_k \hspace{0.05cm}\) is defined as: \(\\[0.25cm]\)

\[ \Gamma(X_k) = \dfrac{\mu_{3}}{\sigma(X_k)^{3}} \]

where:

\[ \mu_{3}\hspace{0.1cm} =\hspace{0.1cm} \frac{1}{n} \cdot \sum_{i=1}^{n} \hspace{0.05cm} x_{ik}^3 \\[0.3cm] \]

Propierties:

Fisher’s skewness coefficient measures the degree of skewness in the distribution of a given statistical variable.

• If \(\hspace{0.12cm} \Gamma(X_k) > 0\) \(\hspace{0.2cm} \Rightarrow \hspace{0.2cm}\) the distribution of \(X_k\) has skewness to the right. \(\\[0.6cm]\)

• If \(\hspace{0.12cm} \Gamma(X_k) < 0\) \(\hspace{0.2cm} \Rightarrow \hspace{0.2cm}\) the distribution of \(X_k\) has skewness to the left. \(\\[0.4cm]\)

3.14 Outliers

There are several definitions of outlier, but here we are going to consider the classic one.

Given a statistical variable \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\), and a sample \(\hspace{0.05cm}X_k \hspace{0.05cm} = \hspace{0.05cm} \left( \hspace{0.01cm} x_{1k} \hspace{0.01cm} , \hspace{0.01cm}x_{2k}\hspace{0.01cm},\dots ,\hspace{0.01cm} x_{nk} \hspace{0.01cm}\right)^t\hspace{0.05cm}\) of that statistical variable.

For any \(\hspace{0.05cm} i\in \lbrace 1,...,n \rbrace\) ,

The observation \(\hspace{0.05cm} x_{ik}\hspace{0.05cm}\) of \(\hspace{0.05cm} \mathcal{X}_k\hspace{0.05cm}\) is an outlier if and only if:

\[x_{ik} \hspace{0.05cm} >\hspace{0.05cm} Q(X_k \hspace{0.05cm} , \hspace{0.05cm} 0.75) + 1.5\cdot IQR(X_k) \hspace{0.5cm}\text{or}\hspace{0.5cm} x_{ik} \hspace{0.05cm} <\hspace{0.05cm} Q(X_k \hspace{0.05cm} , \hspace{0.05cm} 0.25) - 1.5\cdot IQR(X_k) \\\]

where: \(\hspace{0.25cm} IQR(X_k) \hspace{0.12cm} = \hspace{0.12cm} Q(X_k \hspace{0.05cm} , \hspace{0.05cm} 0.75) \hspace{0.08cm} - \hspace{0.08cm} Q(X_k \hspace{0.05cm} , \hspace{0.05cm} 0.25) \hspace{0.25cm}\) is the interquartile range of \(\hspace{0.05cm} X_k \hspace{0.05cm}\).

3.15 Data Matrix

Given \(\hspace{0.05cm} p \hspace{0.05cm}\) statistical variables \(\hspace{0.05cm}\mathcal{X}_1, \mathcal{X}_2, \dots \mathcal{X}_p\hspace{0.05cm}\), and given a sample \(\hspace{0.05cm}X_k = (x_{1k},...,x_{nk})^t\hspace{0.05cm}\) of \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\) for each \(\hspace{0.05cm}k \in \lbrace 1,...,p \rbrace\).

A data matrix of the variables \(\hspace{0.05cm}\mathcal{X}_1,...,\mathcal{X}_1\hspace{0.05cm}\) would be: \(\\[0.35cm]\)

\[ X \hspace{0.05cm}=\hspace{0.05cm} \left( X_1 , X_2,\dots , X_p \right) \hspace{0.05cm}=\hspace{0.05cm} \begin{pmatrix} x_{1}^{t} \\ x_{2} ^t \\ ... \\ x_{n} ^t \end{pmatrix} \hspace{0.05cm}=\hspace{0.05cm} \begin{pmatrix} x_{11} & x_{12}&...&x_{1p}\\ x_{21} & x_{22}&...&x_{2p}\\ ...&...&...&...\\ x_{n1}& x_{n2}&...&x_{np} \end{pmatrix} \\ \]

where:

\(x_i ^t \hspace{0.05cm}=\hspace{0.05cm} \left( x_{i1}, x_{i2}, \dots , x_{ip} \right)\hspace{0.1cm}\) is the vector with the values of the \(\hspace{0.05cm} p \hspace{0.05cm}\) statistical variables \(\hspace{0.05cm}\mathcal{X}_1,\dots ,\mathcal{X}_p\hspace{0.05cm}\) for the \(\hspace{0.05cm}i\)-th element of the sample, for \(\hspace{0.05cm} i \in \lbrace 1,...,n \rbrace\) \(\\[0.4cm]\)

Observations:

\(X \hspace{0.1cm}\) is a matrix with \(\hspace{0.05cm}p\hspace{0.05cm}\) columns and \(\hspace{0.05cm}n\hspace{0.05cm}\) rows, so, is a matrix of size \(\hspace{0.05cm} p\hspace{0.05cm} \text{x}\hspace{0.05cm}n\). \(\\[0.4cm]\)

3.16 Covariance

Given the statistical variables \(\hspace{0.05cm}\mathcal{X}_1, \mathcal{X}_2, \dots \mathcal{X}_p\hspace{0.05cm}\), and given a sample \(\hspace{0.05cm}X_k = (x_{1k},...,x_{nk})^t\hspace{0.05cm}\) of \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\) for each \(\hspace{0.05cm}k \in \lbrace 1,...,p \rbrace\).

The covariance between \(\hspace{0.05cm}X_j\hspace{0.05cm}\) and \(\hspace{0.05cm}X_r\hspace{0.05cm}\) is defined as:

\[ S(X_k, X_r) \hspace{0.1cm}=\hspace{0.1cm} \frac{1}{n} \cdot \sum_{i=1}^{n} \left(\hspace{0.05cm} x_{ik} - \overline{X_k} \hspace{0.05cm}\right)\cdot \left(\hspace{0.05cm} x_{ir} - \overline{X_r} \hspace{0.05cm}\right) \] \(\\[0.4cm]\)

3.17 Covariance Matrix

The covariance matrix of a given data matrix \(\hspace{0.05cm}X \hspace{0.05cm}=\hspace{0.05cm} (X_1,...,X_p)\hspace{0.05cm}\) is: \(\\[0.2cm]\)

\[ S_X = \bigl( \hspace{0.2cm} s_{k,r} \hspace{0.05cm} : \hspace{0.05cm} k,r \in \lbrace 1,...,p \rbrace \hspace{0.2cm} \bigr) \]

where: \(\hspace{0.15cm} s_{k,r} = S(X_k , X_r)\) \(\\[0.25cm]\)

Matrix expression of the covariance matrix :

\[ S_X \hspace{0.1cm} = \hspace{0.1cm} \dfrac{1}{n} \cdot X\hspace{0.1cm}^t \cdot H \cdot X \]

where: \(\hspace{0.15cm} H \hspace{0.1cm}=\hspace{0.1cm} I_n \hspace{0.05cm} - \hspace{0.05cm} \dfrac{1}{n} \cdot 1_{nx1} \cdot 1^t_{nx1} \hspace{0.15cm}\) is the centered matrix

3.18 Correlation

Given the statistical variables \(\hspace{0.05cm}\mathcal{X}_1, \mathcal{X}_2, \dots \mathcal{X}_p\hspace{0.05cm}\), and given a sample \(\hspace{0.05cm}X_k = (x_{1k},...,x_{nk})^t\hspace{0.05cm}\) of \(\hspace{0.05cm}\mathcal{X}_k\hspace{0.05cm}\) for each \(\hspace{0.05cm}k \in \lbrace 1,...,p \rbrace\).

The Pearson linear correlation between the variables \(X_k\) and \(X_r\) is defined as:

\[ r(X_k,X_r) = \frac{S(X_k,X_r)}{S(X_k) \cdot S(X_r)} \] \(\\[0.25cm]\)

3.19 Pearson Correlation Matrix

The Pearson correlation matrix of the data matrix \(X=(X_1 ,..., X_p)\) is : \(\\[0.25cm]\)

\[ R_X =\bigl( \hspace{0.12cm} r_{k,r} \hspace{0.12cm} : \hspace{0.12cm} k,r\in \lbrace 1,...,p \rbrace \hspace{0.12cm} \bigr) \] \(\\[0.25cm]\)

where: \(\hspace{0.2cm} r_{i j} = r(X_i , X_j) \hspace{0.1cm}\) , for \(\hspace{0.12cm} i,j=1,...,p\) \(\\[0.35cm]\)

Matrix expression of the correlation matrix

\[ R_X= D_s^{-1} \cdot S_X \cdot D_s^{-1} \]

where: \[ D_s \hspace{0.05cm} = \hspace{0.05cm} \text{diag} \left( \hspace{0.05cm} \sigma(X_1) ,..., \sigma(X_p) \hspace{0.05cm} \right) \] \(\\[0.5cm]\)

3.20 Absolute Frequency

Given the statistical variables \(\hspace{0.07cm}\mathcal{X}_k\hspace{0.05cm}\), given a sample \(\hspace{0.05cm}X_k = (x_{1k},...,x_{nk})^t\hspace{0.07cm}\) of \(\hspace{0.07cm}\mathcal{X}_k\hspace{0.03cm}\).

3.21 Relative Frequency

Given the statistical variables \(\hspace{0.07cm}\mathcal{X}_k\hspace{0.05cm}\), given a sample \(\hspace{0.05cm}X_k = (x_{1k},...,x_{nk})^t\hspace{0.07cm}\) of \(\hspace{0.07cm}\mathcal{X}_k\hspace{0.03cm}\).

3.22 Cumulative Absolute Frequency

The cumulative absolute frequency of the element \(b\) in \(X_k\) is defined as:

\[ F_{CumA}(b ,X_k) \hspace{0.07cm}= \hspace{0.07cm} F_A \left( \lbrace i=1,...,n \hspace{0.07cm} : \hspace{0.07cm} x_{ik} \leq b \rbrace , X_k \right) \] \(\\[0.4cm]\)

3.23 Cumulative Relative Frequency

The cumulative relative frequency of the element \(b\) in \(X_k\) is defined as:

\[ F_{CumRe}(b,X_k)= \dfrac{F_{CumA}(b,X_k)}{n} \] \(\\[0.4cm]\)

3.24 Frequency Table

A frequency table is a table that contains the absolut, relative and also cumulative frequencies of a statistical variable.

7.1 Visualization Protocol for Quantitative Variables

7.2 Visualization Protocol for Categorical Variables

7.3 Visualization Protocol for Quantitative-Categorical

7.4 Visualization Protocol for Categorical-Categorical

8.1 Estadisticos básicos para las variables cuantitativas

Para las variables cuantitativas:

Netflix_Data.describe()

8.2 Estadisticos básicos para las variables categóricas

Para las variables categóricas (no cuantitativas, en general) :

Netflix_Data.loc[: , ['title', 'description', 'age_certification', 'genres', 'production_countries' ]].describe()

8.3 Gráficos conjuntos para las variables cuantitativas

En esta seccion vamos a hacer un analisis gráfico básico de las variables cuantitativas, consideradas de manera conjunta.

Cargamos las librerias necesarias para los gráficos:

import numpy as np

import seaborn as sns
import matplotlib as mpl
import matplotlib.pyplot as plt

8.3.1 Histograma conjunto de las variables cuantitativas

Vamos a generar un grafico con un histograma para cada una de las variables cuantitativas:

fig, axs = plt.subplots(3, 3, figsize=(11, 11))

p1 = sns.histplot(data=Netflix_Data, x="release_year", stat="proportion", bins=15, color="skyblue", ax=axs[0, 0])
 

p2 = sns.histplot(data=Netflix_Data, x="runtime", stat="proportion", bins=15, color="olive", ax=axs[0, 1])
p2.axes.set(xlabel='runtime', ylabel=' ')
 

p3 = sns.histplot(data=Netflix_Data, x="seasons", stat="proportion", bins=15, color="blue", ax=axs[0, 2])
p3.axes.set(xlabel='seasons', ylabel=' ')
 

p4 = sns.histplot(data=Netflix_Data, x="imdb_score", stat="proportion", bins=15, color="teal", ax=axs[1, 0])
p4.axes.set(xlabel='imdb_score', ylabel=' ')
 

p5 = sns.histplot(data=Netflix_Data, x="imdb_votes", stat="proportion", bins=15, color="purple", ax=axs[1, 1])
p5.axes.set(xlabel='imdb_votes', ylabel=' ')
 

p6 = sns.histplot(data=Netflix_Data, x="tmdb_popularity", stat="proportion", bins=15, color="pink", ax=axs[1, 2])
p6.axes.set(xlabel='tmdb_popularity', ylabel=' ')
 
 
p7 = sns.histplot(data=Netflix_Data, x="tmdb_score", stat="proportion", bins=15, color="red", ax=axs[2, 0])
p7.axes.set(xlabel='tmdb_score', ylabel=' ')
 
fig.savefig('p1.png', format='png', dpi=1200)

plt.show()

Histograma conjunto de las variables cuantitativas

8.3.2 Box-Plot conjunto de las variables cuantitativas

Vamos a generar un grafico con un box-plot para cada una de las variables cuantitativas:

fig, axs = plt.subplots(3, 3, figsize=(11, 11))

p1 = sns.boxplot(data=Netflix_Data, x="release_year", color="skyblue", ax=axs[0, 0])
 

p2 = sns.boxplot(data=Netflix_Data, x="runtime",  color="olive", ax=axs[0, 1])
p2.axes.set(xlabel='runtime', ylabel=' ')
p2.set_xticks( range(int(Netflix_Data['runtime'].min()) , int(Netflix_Data['runtime'].max()) , 100) )
p2.set_yticks( np.arange(0, 1, 0.1)  )

p3 = sns.boxplot(data=Netflix_Data, x="seasons", color="blue", ax=axs[0, 2])
p3.axes.set(xlabel='seasons', ylabel=' ')
 

p4 = sns.boxplot(data=Netflix_Data, x="imdb_score", color="teal", ax=axs[1, 0])
p4.axes.set(xlabel='imdb_score', ylabel=' ')
p4.set_xticks( range(int(Netflix_Data['imdb_score'].min()) , int(Netflix_Data['imdb_score'].max()) , 300) )
p4.set_yticks( np.arange(0, 1, 0.1)  )

p5 = sns.boxplot(data=Netflix_Data, x="imdb_votes", color="purple", ax=axs[1, 1])
p5.axes.set(xlabel='imdb_votes', ylabel=' ')
p5.set_xticks( range(int(Netflix_Data['imdb_votes'].min()) , int(Netflix_Data['imdb_votes'].max()/2) , 500000) )
p5.set_yticks( np.arange(0, 1, 0.1)  )

p6 = sns.boxplot(data=Netflix_Data, x="tmdb_popularity", color="pink", ax=axs[1, 2])
p6.axes.set(xlabel='tmdb_popularity', ylabel=' ')
p6.set_xticks( range(int(Netflix_Data['tmdb_popularity'].min()) , int(Netflix_Data['tmdb_popularity'].max()+1) , 1000) )
p6.set_yticks( np.arange(0, 1, 0.1)  )
 
p7 = sns.boxplot(data=Netflix_Data, x="tmdb_score", color="red", ax=axs[2, 0])
p7.axes.set(xlabel='tmdb_score', ylabel=' ')
p7.set_xticks( range(int(Netflix_Data['tmdb_score'].min()) , int(Netflix_Data['tmdb_score'].max()+1) , 2) )
p7.set_yticks( np.arange(0, 1, 0.1)  )

plt.show()

Box-Plot conjunto de las variables cuantitativas

8.3.3 Empirical-Cumulative-Distribution-Function-Plot conjunto de las variables cuantitativas

Vamos a generar un grafico con un ECDF-plot para cada una de las variables cuantitativas:

fig, axs = plt.subplots(3, 3, figsize=(11, 11))

p1 = sns.ecdfplot(data=Netflix_Data, x="release_year", color="skyblue", ax=axs[0, 0])
p1.set_xticks( range(int(Netflix_Data['release_year'].min()) , int(Netflix_Data['release_year'].max()+20) , 20) )
p1.set_yticks( np.arange(0, 1, 0.1)  )

p2 = sns.ecdfplot(data=Netflix_Data, x="runtime",  color="olive", ax=axs[0, 1])
p2.axes.set(xlabel='runtime', ylabel=' ')
p2.set_xticks( range(int(Netflix_Data['runtime'].min()) , int(Netflix_Data['runtime'].max()) , 100) )
p2.set_yticks( np.arange(0, 1, 0.1)  )

p3 = sns.ecdfplot(data=Netflix_Data, x="seasons", color="blue", ax=axs[0, 2])
p3.axes.set(xlabel='seasons', ylabel=' ')
p3.set_xticks( range(int(Netflix_Data['seasons'].min()) , int(Netflix_Data['seasons'].max()) , 4) )
p3.set_yticks( np.arange(0, 1, 0.1)  )

p4 = sns.ecdfplot(data=Netflix_Data, x="imdb_score", color="teal", ax=axs[1, 0])
p4.axes.set(xlabel='imdb_score', ylabel=' ')
p4.set_xticks( range(int(Netflix_Data['imdb_score'].min()) , int(Netflix_Data['imdb_score'].max()) , 300) )
p4.set_yticks( np.arange(0, 1, 0.1)  )

p5 = sns.ecdfplot(data=Netflix_Data, x="imdb_votes", color="purple", ax=axs[1, 1])
p5.axes.set(xlabel='imdb_votes', ylabel=' ')
p5.set_xticks( range(int(Netflix_Data['imdb_votes'].min()) , int(Netflix_Data['imdb_votes'].max()/2) , 500000) )
p5.set_yticks( np.arange(0, 1, 0.1)  )

p6 = sns.ecdfplot(data=Netflix_Data, x="tmdb_popularity", color="pink", ax=axs[1, 2])
p6.axes.set(xlabel='tmdb_popularity', ylabel=' ')
p6.set_xticks( range(int(Netflix_Data['tmdb_popularity'].min()) , int(Netflix_Data['tmdb_popularity'].max()+1) , 1000) )
p6.set_yticks( np.arange(0, 1, 0.1)  )
 
p7 = sns.ecdfplot(data=Netflix_Data, x="tmdb_score", color="red", ax=axs[2, 0])
p7.axes.set(xlabel='tmdb_score', ylabel=' ')
p7.set_xticks( range(int(Netflix_Data['tmdb_score'].min()) , int(Netflix_Data['tmdb_score'].max()+1) , 50) )
p7.set_yticks( np.arange(0, 1, 0.1)  )

plt.show()

ECDF-Plot conjunto de las variables cuantitativas

8.4 Gráficos conjuntos para las variables categoricas

8.4.1 Bar-plot conjunto de las variables categóricas

Vamos a generar un grafico con un bar-plot para cada una de las variables categóricas, excepto para aquellas cuyo nº de categorias es excesivo, y por tanto no es práctico el gráfico:

fig, axs = plt.subplots(1, 2, figsize=(13, 6))

p1 = sns.countplot(x='type', data=Netflix_Data, ax=axs[0]) 
p1.set_xticklabels(['Movie', 'Show'])
p1.axes.set(xlabel='type', ylabel='count')

p2 = sns.countplot(x='age_certification', data=Netflix_Data, ax=axs[1]) 

plt.show()

Bar-Plot conjunto de variables categoricas